Papers with video encoder
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of understanding multi-modal content, but textonly human-computer interaction is not sufficient for many application scenarios. |
| Approach: | They propose a video-to-text generation task and a multi-modal framework that bootstraps cross-modal training from frozen pre-trained visual & audio encoders and frozen LLMs. |
| Outcome: | The proposed framework can understand both visual and auditory content in video and generate meaningful responses grounded in the visual and audio information presented in the videos. |